Python Programming Tutorials

WebDev tutorial - escaping strings

Is there a way to test why tutorial 16 would cause "must be impossible, not unicode" error after filling out form and hitting submit. If I remove thwart then I can register. I do not know if this is related but when I register not using thwart, the login error shows invalid credentials. Any suggestions on how to debug? Thanks!


class RegistrationForm(Form):
    username = TextField('Username', [validators.Length(min=4, max=20)])
    email = TextField('Email Address', [validators.Length(min=6, max=50)])
    password = PasswordField('New Password', [
        validators.Required(),
        validators.EqualTo('confirm', message='Passwords must match')
    ])
    confirm = PasswordField('Repeat Password')
    accept_tos = BooleanField('I accept the <a href="/about/tos" target="blank">Terms of Service</a> and <a href="/about/privacy-policy" target="blank">Privacy Notice</a> (updated Jan 22, 2015)', [validators.Required()])

@app.route('/register/', methods=['GET', 'POST'])
def register_page():
    try:
        form = RegistrationForm(request.form)
        if request.method == 'POST' and form.validate():
            #flash("register attempted")
            username = form.username.data
            email = form.email.data

            password = sha256_crypt.encrypt((str(form.password.data)))
            c,conn = connection()

            x = c.execute("SELECT * FROM users WHERE username = (%s)",
                (thwart(username)))

            if int(x) > 0:
                flash("That username is already taken, please choose another")
                return render_template('register.html', form=form)

            else:
                c.execute("INSERT INTO users (username, password, email, tracking) VALUES (%s, %s, %s, %s)",
                                (thwart(username),thwart(password),thwart(email), thwart("/introduction-to-python-programming/")))
                conn.commit()
                flash("Thanks for registering!")
                c.close()
                conn.close()
                gc.collect()
                session['logged_in'] = True
                session['username'] = username
                return redirect(url_for('dashboard'))

        return render_template('register.html', form=form)

    except Exception as e:
        return(str(e))
        flash("yay")
        return render_template("register.html", form=form)

You must be logged in to post. Please login or register an account.

Hmm, at first glance, I do not have an answer here.

Compare your code to mine:
https://github.com/PythonProgramming/PythonProgramming.net-Website/blob/master/__init__.py

-Harrison 9 years ago

You must be logged in to post. Please login or register an account.

I am still having issues with the code. Let me see if I understand a fundamental. The following code shows the username so I know I am getting a connection.


@app.route('/register/', methods=["GET", "POST"])
def register_page():

    try:
        form = RegistrationForm(request.form)
        if request.method == "POST" and form.validate():
            username  = form.username.data
            email = form.email.data
            password = sha256_crypt.encrypt((str(form.password.data)))
            c, conn = connection()
            x=username
            return(x)

If, instead I change the return value to a thwarted value it should still return a string right? When I make this simple change I get the error "must be impossible, not unicode"


            x=username
            return(thwart(x))

I can't even get a return value for


return(str(len(thwart(x))))

This tells me my error is inside the thwart. Does that help troubleshoot my problem better? Thanks!

-devtycoon 9 years ago

You must be logged in to post. Please login or register an account.

Well, I would actually say the problem is not with thwart, but with however x is being derived. It's also just highly unlikely something is wrong with the mysqldb escape string function, but highly likely our code is wrong somewhere.

All thwart does is basically find/replace things that could be used for SQL injection. Thus, the string should still exist regardless of whether or not you use thwart. You can just not use for debugging purposes if you like, but that's not where the issue is. Just simply remove thwart and see what x is.

So, x, which is just username, is not being derived, but what about email? Can you return email to see that? Can you thwart the email?
You said you could register without Thwart, but the login credentials didn't work?

Look in the database to see what was put in place for the username upon registration, that might shed some light on what is actually passing through.

One more thing that is missing here is the HTML template for the form, maybe post that too so we can see it.

-Harrison 9 years ago

You must be logged in to post. Please login or register an account.

This is my __init__.py code. I have verified it against github and the tutorials through #17.


from flask import Flask, render_template, redirect, url_for, request, session, flash, g, make_response, send_file
from content_management import Content

from wtforms import Form, BooleanField, TextField, PasswordField, validators
from passlib.hash import sha256_crypt

from MySQLdb import escape_string as thwart

from dbconnect import connection
import gc
# Dictates urls and linkage


app = Flask(__name__)

# list of topics by {TOPIC:["TITLE", "URL"]}
TOPIC_DICT = Content()

@app.route('/')
def main():
    return render_template("main.html")

@app.route('/dashboard/')
def dashboard():
    return render_template("dashboard.html", TOPIC_DICT = TOPIC_DICT)

class RegistrationForm(Form):
    username = TextField('Username', [validators.Length(min=4, max=20)])
    email = TextField('Email Address', [validators.Length(min=6, max=50)])
    password = PasswordField('New Password', [validators.Required(), validators.EqualTo('confirm', message='Passwords must match')])
    confirm = PasswordField('Repeat Password')
    accept_tos = BooleanField('I accept the Terms of Service and Privacy Notice (updated Jan 22, 2015)', [validators.Required()])

@app.route('/register/', methods=['GET', 'POST'])
def register_page():
    try:
        form = RegistrationForm(request.form)

        if request.method == "POST" and form.validate():
            username  = form.username.data
            email = form.email.data
            password = sha256_crypt.encrypt((str(form.password.data)))
            c, conn = connection()

            x = c.execute("SELECT * FROM users WHERE username = (%s)",
                          (thwart(username)))

            if int(x) > 0:
                flash("That username is already taken, please choose another")
                return render_template('register.html', form=form)

            else:
                c.execute("INSERT INTO users (username, password, email, tracking) VALUES (%s, %s, %s, %s)",
                          (thwart(username), thwart(password), thwart(email), thwart("/dashboard/")))

                conn.commit()
                flash("Thanks for registering!")
                c.close()
                conn.close()
                gc.collect()

                session['logged_in'] = True
                session['username'] = username

                return redirect(url_for('dashboard'))

        return render_template("register.html", form=form)

    except Exception as e:
        return(str(e))

@app.route('/login/',methods=['GET','POST'])
def login_page():
    error = ''
    try:

        if request.method == "POST":

            attempted_username = request.form['username']
            attempted_password = request.form['password']

            #flash(attempted_username)
            #flash(attempted_password)

            if attempted_username == "admin" and attempted_password == "%%":
                return redirect(url_for('dashboard'))

            else:
                error = "Invalid credentials. Try Again."

        return render_template("login.html", error = error)

    except Exception as e:
        #flash(e)
        return render_template("login.html", error = error)

@app.errorhandler(404)
def page_not_found(e):
    return render_template("404.html")

@app.errorhandler(405)
def method_not_found(e):
    return render_template("405.html")

@app.errorhandler(500)
def page_not_found(e):
    return ("Ouch, looks like we're knocked out"), 500

if __name__ == "__main__":
    app.run()

I then comment my code line by line in register_page() function, from the bottom, until the error does not appear anymore. The error occurs when I define x. So I then comment x out, and begin testing each variable, username, email and password. I even tested a raw string where x = "/dashbaord/".

I debug using the following replacement for x, and expect that if the code works I will get a string appearing on a blank page showing "So Far So Good". Unfortunately that is not the case. From oracle's documentation on escape_string, they claim it will result in a string. That tells me, If I understand the syntax, I should be able to return(thwart("/dashbaord/")) and get "/dashbaord/" back on a blank screen.


            x = email
            x = thwart(x)

            return('So far so good')

return(thwart(username)) ----->results in the error-------> "must be impossible, not unicode"
return(thwart(password)) ------>results in the error-------> "must be impossible, not str"
return(thwart(email)) ------------>results in the error-------> "must be impossible, not unicode"

When I wrap the variables in a str() function, the error shows  "must be impossible, not str" for every single variable

This is telling me that the data passed through username and email variable is unicode and the data passed into the password is a string.  Database is showing strings are being stored.

Everything I am reading online is showing that your syntax is correct!  Just playing devils advocate, I started the tutorial over from the beginning and the only difference that I can see is the version of linux.  Is there a chance mySQL needs to be reinstalled?

Thanks for the help Harrison

I am using:
Ubuntu 15.10
Python 2.7.10

Resources
http://mysql-python.sourceforge.net/MySQLdb.html#connection-objects
https://docs.oracle.com/cd/E17952_01/connector-python-en/connector-python-api-cext-escape-string.html

Current Packages installed on python:

['blinker==1.4', 'flask-mail==0.9.1', 'flask-mysqldb==0.2.0', 'flask-wtf==0.12',                                                                                                       'flask==0.10.1', 'itsdangerous==0.24', 'jinja2==2.8', 'markupsafe==0.23', 'oaut                                                                                                      hlib==1.0.3', 'passlib==1.6.5', 'requests-oauthlib==0.5.0', 'tweepy==3.4.0', 'vi                                                                                                      rtualenv==13.1.2', 'werkzeug==0.10.4', 'wtforms==2.0.2']

-devtycoon 9 years ago

Last edited 9 years ago

You must be logged in to post. Please login or register an account.

I would like to put my hand up and say I'm having the exact same problem. the more i search the more noob i feel... python3 has the same (must be impossible) issue so i dont think it's python... is there a way to downgrade MySQLdb? could that be the cause? sorry if my suggestions are silly

-Thomas Ibbotson 9 years ago

You must be logged in to post. Please login or register an account.

I may be confused, but what exactly makes you feel as if it is thwart? Thwart simply finds/replaces. Does your application run just fine, minus the use of thwart? I would not run a production server without escaping user input, but for testing on a dev server, if you remove thwart, then that would theoretically solve the problem if the issue was indeed thwart.

If OP removes Thwart, it still turns out to be an error for him, which tells me the issue is not thwart, but it is elsewhere. I was told if he removed thwart then he could register, just not log in again due to invalid credentials (but I wonder if that'd be true if you removed thwart on that input too.)

Curious to know what value is put into the database when not using thwart.

edit:

Also, go ahead and remove the thwart import entirely, and all uses of it. Try to see if things work as intended without using it at all. Remove it from the imports, so you get a 500 just in case you unknowingly use it in a place you forgot about or something.

I just would be super surprised if thwart really was the cause here, but this is also pretty coincidental to have two people suddenly have this issue when I have never heard of it, nor has Google search :P

-Harrison 9 years ago

Last edited 9 years ago

You must be logged in to post. Please login or register an account.

Hmm. Are you totally confident you are indeed using Python 2 and not Python 3? This string vs unicode nonsense is something that would happen in Py 3.

Also, let's revisit the scenario when not using thwart at all. Let's consider back when you registered without thwart, a string is slapped into the database. Does that string for the username appear to be reasonable, as in what you typed?

When you go to re-log in, you are saying that the credentials do not match. Are you also removing the thwarting being applied to both the username and password on both registration and login? If not, then the password and probably usernames would not match.

I am just hoping to see some sort of for sure confirmation that the problem is indeed with thwart. I propose you remove all instances of thwart, and test the process (doing this in a dev environment only).

If the process works as expected, without using thwart at all (remove the thwart import is my suggestion, so a 500 is thrown if you attempt to use it unknowingly), then we can know for sure it is in the thwart.

If the process still doesn't work, then it's not thwart causing the problem.

-Harrison 9 years ago

You must be logged in to post. Please login or register an account.

I was playing around with the c.execute and I got all sorts of errors. i removed the thwart and now get an error: "not all arguments converted during string formatting" from the str(e)

@app.route('/register/', methods=['GET', 'POST'])
def register():

    try:
        form = RegistrationForm(request.form)
        if request.method == 'POST' and form.validate():
            #flash("register attempted")

            username = form.username.data
            email = form.email.data

            password = sha256_crypt.encrypt((str(form.password.data)))
            c,conn = connection()

            x = c.execute("SELECT * FROM users WHERE username = (%s)", 
                (username))

            if int(x) > 0:
                flash("That username is already taken, please choose another")
                return render_template('register.html', form=form)

            else:
                c.execute("INSERT INTO users (username, password, email, tracking) VALUES (%s, %s, %s, %s)",
                    (username, password, email, "/introduction-to-python-programming/"))
                conn.commit()
                flash('Thanks for registering')
                c.close()
                conn.close()
                gc.collect()
                #session['logged_in'] = True
				#session['username'] = username
                return redirect(url_for('intro_to_py'))
				
        gc.collect()
        #flash("hi there.")
        return render_template('register.html', form=form)
    except Exception as e:
        return(str(e))

-Thomas Ibbotson 9 years ago

You must be logged in to post. Please login or register an account.

when playing around with the c.execute line to do only a simple query, i get:

(1054, "Unknown column 'Tibbo' in 'where clause'")

I also couldn't use:

c.execute("SELECT * FROM users WHERE username = (%s)", (username))

This gives me the "not all arguments converted during string formatting" error...

I tried to use:

c.execute("SELECT * FROM users WHERE username = %s" % username)

to get the 1054 error...

-Thomas Ibbotson 9 years ago

You must be logged in to post. Please login or register an account.

Hi Harrison

For the record, the reason I can not login with valid credentials is because tut 17 still has admin and password as only valid login. I am a noob sorry. I can prove that something is going on with MySQLdb, unless you can show me an error in my two lines of code.

If I could start my post over again I would just ask this question.
On your machine does this code work on your machine?


>>> import MySQLdb
>>> MySQLdb.escape_string("'")
"'"

I found that on stack.
http://stackoverflow.com/questions/2561178/python-equivalent-of-mysql-real-escape-string-for-getting-strings-safely-into-m

It says I should be able to open python , import MySQLdb, then escape the string "'"

When I run that full two lines of code I get the error in the python console (the console is accessed through the cloud server I rent from digital ocean):


root@dev:~# python
Python 2.7.10 (default, Oct 14 2015, 16:09:02)
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>> MySQLdb.escape_string("'")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: must be impossible<bad format char>, not str

-devtycoon 9 years ago

Last edited 9 years ago

You must be logged in to post. Please login or register an account.

Hmm, another thing is you are using Python 2.7.10, which is very new. I am using 2.7.8.

Also:

MySQL-python==1.2.3

You could do something like:

pip install MySQL-python==1.2.3